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Supervisory  Control  of  Multiple  Uninhabited  Systems 
Methodologies  and  Human-Robot  Interface  Technologies 

(STO-MP-  HFM-217) 

Executive  Summary 

The  HFM-217  Workshop  on  “Supervisory  Control  of  Multiple  Uninhabited  Systems  -  Methodologies  and  Human- 
Robot  Interface  Technologies”  took  place  in  Prague,  Czech  Republic,  from  Tuesday  8th  May  through  Thursday  10th 
May  2012.  The  technical  part  of  the  program  consisted  of  2  keynote  presentations,  13  interactive  Technology 
Demonstrations  followed  by  an  overall  summary/award  presentation,  a  breakout  session  in  three  groups  on  Priority 
Research  Challenges  followed  by  a  plenary  presentation  of  the  results  and  a  subsequent  consolidation  of  these,  and  a 
closing  session.  Approximately  50  researchers  participated  in  this  workshop. 

The  primary  purposes  of  this  invitation-only  workshop  were: 

•  To  review  the  recent  advances  in  the  areas  of  Multi-Uninhabited  Military  Vehicle  (UMV)  supervisory 
control  methodologies  and  operator  interface  technologies,  and 

•  To  identify  the  next  steps  and  critical  research  needed  to  achieve  future  NATO  visions  for  multi-UV 
employment,  and 

•  To  exchange  knowledge  and  ideas  with  fellow  researchers. 

The  thirteen  working  demonstrators  from  HFM-170  that  were  brought  together  in  one  meeting  place  formed  the  core 
of  this  capstone  workshop.  The  idea  of  putting  them  side-by-side  was  brilliant  and  created  at  atmosphere  in  which 
discussions  were  open,  constructive,  and  hands-on,  and  that  inspired  the  collective  thinking  about  the  new  challenges 
that  lie  ahead  of  us. 

A  very  well  organized  and  extremely  interesting  workshop  developed.  After  the  interactive  technology  demonstrator 
sessions,  three  breakout  groups  were  formed  to  respond  to  three  questions: 

•  What  is  the  highest  priority  research  needs  to  enable  future  NATO  UV  visions? 

•  Who  are  the  key  researchers/laboratories  in  this  area  across  NATO? 

•  What  are  future  CON OPS  to  explore,  and  what  are  the  most  promising  interface  technologies? 

The  results  were  coalesced  into  a  new  candidate  model  of  supervisory  control  that  considers  all  high  priority  research 
facets. 

There  are  two  main  sets  of  conclusions  that  can  be  drawn  from  this  workshop  and  that  can  also  be  read  as 
recommendations : 

•  The  first  conclusion  is  that  models  need  to  be  developed  and  selected  that  can  support  the  human  factors 
engineers  balancing  act  of  deciding  which  type  of  task  distribution  in  controlling  multiple  uninhabited 
vehicles  best  matches  the  real-world  drivers.  A  model  like  that  could  also  be  instrumental  in  providing 
consistency  in  user  interface  design  decisions  around  NATO. 

•  The  second  conclusion  is  based  on  the  observation  that  the  ongoing  evolution  of  uninhabited  systems 
reveals  a  rapid  blurring  of  the  boundaries  between  the  roles  of  humans  and  machines  as  well  as  the 
difference  between  the  notions  of  uninhabited  systems  and  other  automated  process.  Simply  put,  in  the 
future  we  need  not  focus  much  more  on  Supervising  Uninhabited  Systems,  but  rather  on  the  multi-faceted 
and  reciprocal  interaction  between  human  and  automated  processes.  That  a  particular  system  is  uninhabited 
will  be  a  disappearing  notion.  What  we  need  to  have  is  flexible/adjustable,  trustworthy  automation  being 
operated  under  user  sovereignty.  This  will  likely  require  not  just  intense  collaboration  but  potentially  even  a 
merge  of  the  field  of  human  factors  engineering  with  other  engineering  fields  such  as  process  automation 
and  computer  science. 
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Commande  et  surveillance  de  multiples  systemes  sans  pilote 
Methodologies  et  technologies  d’interfaces  homme-robot 

(STO-MP-HFM-2 1 7) 

Synthese 

L’atelier  HFM-217  «  Commande  et  surveillance  de  multiples  systemes  sans  pilote  -  Methodologies  et  technologies 
d’interfaces  homme-robot  »  a  ete  organise  a  Prague  en  Republique  tcheque,  du  mardi  8  mai  aujeudi  10  mai  2012.  La 
partie  technique  du  programme  a  consiste  en  2  interventions  d’interet  majeur,  13  demonstrations  interactives  de 
technologies  suivies  d’une  synthese  d’ ensemble  et  d’une  remise  de  prix,  une  session  divisee  en  trois  groupes  de 
discussion  traitant  des  problemes  des  recherches  prioritaires  suivie  d’une  presentation  pleniere  des  resultats  qui  ont 
ensuite  ete  regroupes,  puis  d’une  session  de  cloture.  Cet  atelier  a  refu  la  participation  d’environ  cinquante  chercheurs 
invites. 

Les  principaux  objectifs  de  HFM-217  etaient: 

•  l’examen  des  progres  recents  dans  le  domaine  des  methodologies  de  commande  et  de  surveillance  et  celui 
des  technologies  d’interfaces  operateur  pour  de  multiples  vehicules  militaires  sans  pilote  (UMV) ; 

•  1’  identification  des  prochaines  etapes  et  des  recherches  essentielles  necessaires  a  la  realisation  des  ambitions 
de  l’OTAN  vis-a-vis  d’une  utilisation  multi-UV  ;  et 

•  l’echange  de  connaissances  et  d’idees  entre  chercheurs. 

Les  treize  modeles  fonctionnels  de  demonstration  issus  de  l’HFM-170  avaient  ete  regroupes  sur  un  seul  lieu  de 
reunion  et  ont  constitue  le  coeur  de  cet  atelier  fondamental.  Les  disposer  les  uns  a  cote  des  autres  s’est  avere  etre  une 
excellente  idee  qui  a  genere  une  ambiance  propice  a  des  discussions  ouvertes,  constructives  et  pratiques  ;  cela  a 
inspire  une  reflexion  collective  sur  les  nouveaux  defis  qui  nous  attendent. 

Un  atelier  tres  bien  organise  et  extremement  interessant  s’en  est  suivi.  Apres  les  sessions  de  demonstration  des 
technologies  interactives,  trois  groupes  de  discussion  ont  ete  constitues  pour  repondre  a  trois  questions  : 

•  Quelle  est  la  priorite  la  plus  importante  pour  que  la  recherche  puisse  permettre  de  progresser  vers  les  visions 
futures  de  l’OTAN  en  matiere  d’UV  ? 

•  Quels  sont  les  chercheurs/laboratoires  les  plus  en  pointe  dans  ce  domaine  au  sein  de  LOT  AN  ? 

•  Quels  sont  les  futurs  concepts  d’operation  (CONOPS)  a  explorer,  et  quelles  sont  les  technologies 
d’interface  les  plus  prometteuses  ? 

Les  resultats  ont  ete  reunis  en  un  nouveau  projet  de  modele  de  commande  et  surveillance  qui  tient  compte  de  tous  les 
aspects  de  recherche  prioritaire. 

Deux  conclusions  principals  peuvent  etre  tirees  de  cet  atelier  et  servir  de  recommandations  : 

•  La  premiere  reflete  la  necessity  de  developper  et  selectionner  des  modeles  qui  peuvent  aider  les  ergonomes 
a  equilibrer  le  choix  du  type  de  distribution  des  taches  de  controle  de  multiples  vehicules  sans  pilote  qui 
correspond  le  mieux  a  des  pilotes  reels.  Un  tel  modele  pourrait  egalement  contribuer  a  la  coherence  des 
decisions  dans  LOT  AN  en  matiere  de  conception  des  interfaces  utilisateurs. 

•  La  deuxieme  est  basee  sur  une  observation  qui  soidigne  que  revolution  continuelle  des  systemes  sans  pilote 
brouille  en  fait  rapidement,  d’une  part,  la  frontiere  entre  le  role  de  Fhomme  et  celui  de  la  machine,  et 
d’autre  part,  la  difference  entre  la  notion  de  systemes  sans  pilote  et  celle  d’autres  processus  automatises.  En 
bref,  il  n’est  pas  necessaire  de  nous  concentrer  davantage  sur  la  surveillance  des  systemes  sans  pilote  mais 
plutot  sur  les  interactions  multiformes  et  reciproques  entre  l’homme  et  les  processus  automatises.  Dire 
qu’un  systeme  particular  est  sans  pilote  est  une  notion  vouee  a  disparaitre.  Ce  a  quoi  nous  devons  parvenir 
est  un  systeme  automatise  Liable,  souple  ou  ajustable  qui  fonctionne  sous  la  responsabilite  de  son  operateur. 
Une  simple  collaboration  intensive  risque  de  ne  pas  suffire  et  une  fusion  sera  probablement  necessaire  entre 
le  domaine  ergonomie  et  d’autres  domaines  d’ingenierie,  comme  l’automatisation  des  processus  ou 
l’informatique. 
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ABSTRACT 

About  50  human  factors  researchers  gathered  in  Prague  in  May  2012  for  a  workshop  to  discuss  human 
factors  aspects  of  uninhabited  systems.  More  specifically,  they  focussed  on  supervisory  control  of  multiple 
uninhabited  systems.  The  workshop  was  strongly  enriched  by  the  presence  of  13  interactive  technology 
demonstrators  from  the  different  nations.  Lively  and  inspiring  discusssions  ensued  and  the  collective 
thinking  and  insight  was  raised  to  new  levels.  Two  important  conclusions  can  be  drawn  from  this  workshop. 
First,  there  is  a  need  of  adequate  models  that  can  support  engineers  (during  design)  and  operators  (during 
missions)  to  decide  which  type  of  task  distribution  in  controlling  multiple  uninhabited  vehicles  best  matches 
the  real-world  needs  and  constraints.  This  needs  to  be  developed.  Second,  we  need  not  focus  much  more  on 
Supervising  Uninhabited  Systems  as  such,  but  rather  need  to  turn  our  attention  to  the  multi-faceted  and 
reciprocal  interaction  between  human  and  automated  processes.  That  a  particular  system  is  uninhabited  will 
simply  be  a  disappearing  notion.  What  we  need  to  have  is  flexible,  adjustable,  trustworthy  automation  being 
operated  under  user  sovereignty.  This  will  likely  require  not  just  intense  collaboration  but  potentially  even  a 
merge  of  the  field  of  human  factors  engineering  with  other  engineering  fields  such  as  process  automation 
and  computer  science. 


KEYWORDS 

Uninhabited  Military  Vehicles,  Unmanned  Vehicles,  Supervisory  Control,  Human-Machine  Symbiosis, 
Human-Robot  Interaction,  Cooperative  Automation,  Mixed  Teams,  Advanced  Operator  Interfaces, 
Swarms 


1.  INTRODUCTION 
1.1  Purpose  and  Scope 

The  history  of  the  group  of  scientists  and  engineers  that  organised  this  workshop  dates  back  to  1988  when 
EOARD  sponsored  conferences  on  the  topic  of  a  “ Human-Electronic  Crew”.  After  that,  NATO  RTO 
followed  with  the  HFM-078  taskgroup  on  “ Uninhabited  Military  Vehicles:  Human  Factors  in  Augmenting 
the  Force”.  They  organised  a  workshop  in  Leiden,  The  Netherlands  in  2003,  and  also  a  concluding 
symposium  in  Biarritz,  France  in  2006  entitled  “ Human  Factors  of  Uninhabited  Military  Vehicles  as  Force 
Multipliers”  (HFM-135).  After  that,  the  group  continued  as  taskgroup  HFM-170  “ Supervisory  Control  of 
Multiple  Uninhabited  Systems  -  Methodologies  and  Enabling  Human-Robot  Interface  Technologies.  With 
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this,  their  center  of  attention  shifted  forward  towards  the  challenges  of  supervising  multiple  uninhabited 
systems  by  a  single  operator.  The  focus  of  their  work  was  on  creating  high-fidelity  technology 
demonstrators,  when  possible  as  a  collaborative  effort  between  the  nations.  These  13  demonstrators  from 
HFM-170  brought  together  in  one  place  formed  the  core  of  this  capstone  workshop.  The  workshop 
announcement  text  can  be  found  in  Appendix  A. 

The  primary  purposes  of  this  Workshop  were 

(1)  To  review  the  recent  advances  in  the  areas  of  multi-Uninhabited  Military  Vehicle  (UMV) 
supervisory  control  methodologies  and  operator  interface  technologies,  and 

(2)  To  identify  the  next  steps  and  critical  research  needed  to  achieve  future  NATO  visions  for 
multi-UMV  employment,  and 

(3)  To  exchange  ideas  with  fellow  researchers. 

1.2  Symposium  Program  &  Participation 

The  HFM-217  Workshop  on  “ Supervisory  Control  of  Multiple  Uninhabited  Systems  -  Methodologies  and 
Human-Robot  Interface  Technologies ”  took  place  in  Praque,  Czech  Republic  from  Tuesday  8th  May  through 
Thursday  10th  May  2012.  The  technical  part  of  the  program  consisted  of  2  keynote  presentations,  13 
interactive  Technology  Demonstrations  followed  by  an  overall  summary,  a  breakout  session  in  three  groups 
on  Priority  Research  Challenges  followed  by  a  plenary  presentation  of  the  results  and  a  subsequent 
consolidation  of  these,  and  a  closing  session  which  included  the  TER  summary  presentation  of  which  the 
current  paper  is  the  written  version.  The  detailed  program  can  be  found  in  Appendix  B.  About  46 
participants  from  a  range  of  nations  and  affiliations  participated  in  the  -  invitation  only  -  workshop. 


2.  OBSERVATIONS 

My  observations  are  presented  below  in  three  separate  sections  reflecting  the  overall  structure  of  the 
workshop. 

2.1  Overall  Impression 

The  idea  of  bringing  all  13  Technology  Demonstrators  together  in  one  place  was  brilliant  and  created  an 
atmosphere  in  which  discussions  were  open,  constructive,  and  hands-on,  and  that  inspired  the  collective 
thinking  about  the  new  challenges  that  lie  ahead  of  us.  A  very  well  organized  and  extremely  interesting 
workshop  developed.  The  organisers  certainly  achieved  their  goals.  It  must  also  be  said  that  both  keynote 
presentations  contributed  strongly  to  the  sense  of  relevance  of  the  topic,  both  in  scientific  terms  (Chris 
Miller)  as  well  as  in  operational  terms  (Jeffery  Eggers). 

2.2  Technology  Demonstrations 
2.2.1  Process 

The  13  Technology  Demonstrators  are  described  in  detail  in  a  publication  by  HFM-170.  During  the 
workshop  they  were  presented  briefly  in  a  plenary  session.  After  that,  all  participants  moved  to  a  hall  in 
which  the  demonstrators  had  been  set  up.  They  were  led  in  small  groups  in  round-robin  style  around  the 
demonstrators  in  such  a  way  that  everyone  got  to  experience  all  demonstrators.  This  led  to  a  very  intense 
experience,  a  hall  vibrant  with  activity. 
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Taken  together,  the  set  of  demonstrators  displayed  a  great  mixture.  Some  reflected  a  single  study  where 
others  could  be  seen  as  the  culmination  of  decades  of  work  of  many  people.  Some  demonstrators  had  been 
used  in  field  trials  or  even  in  large  collaborative  trials  whereas  others  were  laboratory  studies.  And  some 
demonstrators  explored  new  or  future  interaction  approaches  while  others  performed  formal  lab  evaluations 
of  different  existing  interaction  designs. 

The  similarity  between  these  Supervisory  Control  demonstrators  was  that  all  of  them  (1)  took  the  viewpoint 
of  the  operator  work  environment  and  (2)  dealt  with  positioning  (multiple)  vehicles  and/or  observation  tasks. 
This  is  obviously  a  reflection  of  the  typically  usage  of  UMVs  now  and  in  the  near  future.  However,  there  are 
other  viewpoints  and  tasks  that  need  Human  Factors  attention.  For  example,  what  knowledge  do  we  need  to 
support  the  supervisor  of  a  future  autonomous  fuel  transport  convoy?  What  do  we  need  to  be  able  to  support 
soldiers  that  -  in  theatre  -  encounter  UMVs  (or  swarms  of  them)  that  are  supervised  by  someone  elsewhere? 
And  what  are  the  human  factors  implications  of  supervising  UMVs  or  multiple  UMVs  that  perform  actions 
beyond  moving  and  observation  tasks,  such  as  weapon  usage  or  target  designation?  Certainly  these  topics 
need  to  be  considered  too. 

The  feedback  forms  which  were  handed  out  during  the  demonstration  tour  were  used  to  summarise  the  event, 
an  effort  that  was  elegantly  carried  out  by  Mr.  Harry  Funk.  A  group  voting  procedure  was  also  part  of  the 
process.  Awards  were  given  for  scientific  achievement  (GER-1),  near-term  impact  (US-1)  and  innovation 
(FRA-1). 

2.2.2  Distributing  Control 

The  main  topic  of  this  workshop  is  supervisory  control  and  it  is  therefore  relevant  to  look  at  the  way  in 
which  the  different  demonstrators  had  chosen  to  distribute  control  between  human  and  machine.  To  be  able 
to  investigate  this,  I  plotted  all  13  demonstrators  on  two  axes  in  Figure  1.  The  horizontal  axis  describes  at 
what  point  in  time  the  decision  was  made  concerning  the  distribution  of  control  between  human  and  machine 
for  that  specific  demonstrator.  This  when- axis  has  three  categories:  the  task  distribution  was  either  fixed  at 
design  time,  set  during  mission  preparation,  or  adjustable  during  mission  execution.  The  vertical  axis 
describes  the  type  of  entity  that  drove  the  task  distribution.  This  driver- axis  has  four  categories:  task-based, 
depended  on  operator  resources,  driven  by  operator  attention,  or  depended  on  the  abilities  of  the  operator 
(skills)  and  the  availability  of  machine  autonomous  behaviours.  The  demonstrators  could  easily  be  plotted  in 
this  matrix,  see  Figure  1  below. 

Many  demonstrators  showed  a  task  distribution  approach  that  simply  assigned  certain  types  of  tasks  to  the 
operator  and  other  types  of  tasks  to  an  automated  process  (first  row  in  Figure  1).  Tasks  were  systematically 
distinguished  using  different  task  models,  such  as  OODA-loop  or  State2Campaign.  In  two  of  these  cases 
(FRA2  and  PT)  the  choice  between  human  and  machine  was  made  during  mission  preparation  time  (i.e.,  was 
fixed  for  the  mission). 

In  three  cases  (US1,  US4,  GER)  the  task  distribution  was  flexible  during  the  mission  and  driven  by  operator 
resources  (second  row).  In  the  two  US  demonstrators  it  was  basically  left  to  the  operator  to  take  control 
whenever  s/he  had  time  or  felt  it  was  needed;  the  GER  demonstrator  featured  an  elaborate  workload  model 
that  used  estimators  and  measurements  to  optimally  redistribute  tasks. 
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Figure  1:  How  is  control  distributed? 


Much  more  can  be  said  about  this,  but  for  here  it  is  interesting  to  note  that  the  demonstrators  (or  really:  their 
human-machine  interfaces)  can  be  categorised  easily  and  insightfully  using  these  two  dimensions.  Beyond 
that,  however,  it  is  very  relevant  to  note  that  in  setting  up  these  demonstrators  limited  systematic  attention 
had  been  given  to  the  particular  choice  of  task  distribution  for  each  individual  demonstrator.  Obviously,  for 
actual  operational  systems  such  choices  will  typically  be  driven  by  real-world  drivers  such  as  cost, 
performance,  acceptance,  reliability,  etc...  However,  to  translate  these  real-world  drivers  systematically  and 
explicitly  into  decisions  about  how  and  when  tasks  should  be  distributed  between  human  and  automated 
process  in  controlling  multiple  uninhabited  vehicles  is  not  an  easy  task,  and  requires  the  development  and 
selection  of  appropriate  models. 

2.3  Highest  Priority  Research  Challenges 
2.3.1  Process 

After  the  interactive  technology  demonstrator  sessions,  three  breakout  groups  were  formed  to  respond  to 
three  questions: 

•  What  are  the  highest  priority  research  needs  to  enable  future  NATO  UMV  visions? 

•  Who  are  the  key  researchers/laboratories  in  this  area  across  NATO? 

•  What  are  future  CON OPS  to  explore,  and  what  are  the  most  promising  interface  technologies? 

The  amount  of  ideas  and  collective  insights  was  overwhelming.  All  results  were  presented  plenary  and  the 
research  priorities  were  then  coalesced  by  Prof.  Gilles  Coppin  into  a  new  candidate  model  of  supervisory 
control  that  considers  all  high  priority  research  facets.  This  model  puts  the  OODA-loop  into  the  centre  and 
groups  all  research  topics  around  it  in  four  blocks:  people,  architecture,  automation,  and  interaction.  This 
model  certainly  deserves  to  be  worked  out  in  more  detail  in  an  effort  to  help  us  all  categorise  the  elements  of 
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our  common  research  future.  The  workshop  did  not  attempt  to  prioritise  the  material  due  to  lack  of  time  and 
guidelines  and  also  due  the  overwhelming  scope  and  volume  of  the  material  that  was  collected. 

2.3.2  Human-Machine  Symbiosis 

The  range  of  research  ideas  and  conops  that  was  brought  to  the  table  was  huge.  For  instance,  for  interface 
aspects  someone  recapitulated  it  as  the  need  to  have  distributed,  collaborative,  hybrid,  service-oriented,  ad 
hoc,  hand-held  [or  even  body-integrated]  types  of  control  of  partly  autonomous  vehicles  and  their  payloads 
and  just  about  everything  else  too.  Mention  was  made  of  future  robots  that  would  necessarily  control  humans 
as  much  as  the  other  way  around;  of  adversaries  that  control  robots  possibly  including  our  own;  of 
autonomous  schoolbuilding  robots;  or  even  of  the  ultimate  “Please  win  this  part  of  the  war  for  me”-robots.  It 
was  also  envisaged  that  autonomous  behaviours  (military  or  not)  will  become  omni-present  in  public 
environments  as  well  as  on  the  battle  field.  It  was  pointed  out  that  in  the  near  future  open  access  to  billions  of 
live  civil  HD  sensors  streams  radiated  from  mobile  phones  and  the  like  might  become  a  game  changer  in  the 
military  theatre  as  well,  pushing  the  observation  roles  of  UMVs  into  other  niches. 

To  be  able  to  report  on  this  spectrum  of  results  I’ve  looked  at  them  from  the  perspective  of  a  symbiosis 
between  human  and  machine.  I’ve  taken  this  viewpoint  because  although  we  do  speak  of  supervisory 
control,  which  implies  hierarchy,  everything  that  was  said  during  the  session  suggests  a  future  in  which  we 
are  much  better  off  approaching  the  interaction  between  human  and  machine  from  both  sides. 

From  biology  we  can  learn  that  for  any  symbiosis  to  work,  one  needs  to  consider  the  relationship  between 
the  parties  involved,  the  context  or  environment  in  which  it  takes  place,  and  the  desired  outcomes  of  the 
symbiosis.  In  Figure  2  below  I’ve  listed  a  high-level  summary  of  the  research  elements  that  came  forward  in 
the  session.  Under  the  heading  “supervisory  control  relationship”  you  find  topics  such  as  natural  interaction, 
mutual  trust,  etc.  For  instance,  the  topic  “mutual  state  knowledge”  means  that  for  optimal  collaboration  both 
the  supervisor  and  the  supervised  need  to  be  aware  of  the  current  state  of  the  other.  It  makes  sense  for  a 
decision  support  tool  to  have  an  estimate  of  the  commander’s  current  knowledge  of  the  mission  progress  just 
as  well  as  it  makes  sense  for  the  commander  to  know  whether,  for  example,  a  controlled  autonomous  fuel 
convoy  is  capable  to  self-sustain  its  mission.  Under  the  heading  “environmental  context”  we  see  a  topic  like 
social  interaction  which  is  meant  to  denote  the  need  for  future  UMVs  to  develop  the  capability  to  interact 
socially  with  the  soldiers  in  the  field,  or  possibly  even  with  the  citizens  in  the  villages.  Under  the  heading 
“Process  Output”  there  is  just  one  topic  which  has  been  put  there  as  an  example.  It  denotes  the  need  for  a 
close  symbiosis  with  automated  systems  including  UMVs  in  order  to  leverage  the  information  gathering  and 
processing  needed  to  achieve  decision  superiority. 
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Figure  2:  Some  critical  success  factors  for  human-machine  symbiosis. 

In  much  of  the  above  the  old  differences  between  the  roles  of  humans  and  machines  as  well  as  the  difference 
between  the  notions  of  uninhabited  systems  and  other  automated  processes  is  absent.  Simply  put,  in  the  next 
future  we  need  not  focus  much  more  on  Supervising  Uninhabited  Systems  but  rather  on  the  multi-faceted 
and  reciprocal  interaction  between  human  and  automated  processes.  That  a  particular  system  is  uninhabited 
will  be  a  disappearing  notion.  It  will  be  treated  like  any  other  automated  process,  like  image  analysis.  This 
will  have  a  profound  influence  on  the  way  we  need  to  approach  these  interactions.  No  longer  can  this  be 
approached  from  a  human  factors  engineering  point  of  view.  Rather,  we  need  to  melt  human  factors 
engineering  together  with  other  engineering  fields  such  as  process  automation  and  computer  science  to  be 
able  to  create  those  systems  that  are  needed  for  NATOs  future  UMV  operations. 


3.  CONCLUSIONS  &  RECOMMENDATIONS 

An  important  conclusion  I’d  like  to  draw  is  that  putting  in  the  effort  of  bringing  a  large  number  of  working 
demonstrators  from  different  nations  together  in  one  meeting  place  is  very  rewarding.  It  stimulates  the 
discussion  and  thought  exchange  tremendously  and  creates  a  very  vivid  working  atmosphere.  Even  though 
the  cost  turned  out  to  be  prohibitive  in  some  cases  for  bringing  everything  that  was  envisioned,  the  gist  was 
always  conveyed  easily,  even  with  simpler  means.  I  recommend  the  use  of  this  approach  more  often. 

There  are  two  main  sets  of  scientific  conclusions  that  can  be  drawn  from  this  workshop  and  that  can  also  be 
read  as  recommendations. 

1.  The  first  conclusion  relates  to  the  words  supervisory  control  in  the  title  of  the  workshop  and  became 
apparent  after  experiencing  the  13  demonstrators  and  the  associated  research  programs.  It  turned  out 
that  for  each  of  these  demonstrators  implicit  decisions  had  been  made  about  the  particular  distribution 
of  control  between  human  and  machine.  Along  the  w/ze«-dimension  the  task  distribution  was  either 
fixed  at  design  time,  set  during  mission  preparation,  or  adjustable  during  mission  execution.  The 
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elements  that  drove  the  task  distribution  were  either  task-based,  depended  on  operator  resources,  were 
driven  by  operator  attention,  or  were  depended  on  the  abilities  of  the  operator  (skills)  and  the 
availability  of  machine  autonomous  behaviours.  All  demonstrators  (and  therefore  all  their  human- 
machine  interfaces)  could  be  mapped  onto  these  two  dimensions,  which  is  interesting  in  itself. 
However,  it  also  became  clear  that  limited  systematic  attention  had  been  given  to  the  particular  choice 
of  task  distribution  for  each  individual  demonstrator.  Obviously,  for  actual  systems  such  choices  will 
typically  be  driven  by  real-world  drivers  such  as  cost,  performance,  acceptance,  reliability,  etc...  The 
first  conclusion  is  therefore  that  models  need  to  be  developed  and  selected  that  can  support  the  human 
factors  engineers  balancing  act  of  deciding  which  type  of  task  distribution  in  controlling  multiple 
uninhabited  vehicles  best  matches  the  real-world  drivers.  A  model  like  that  could  also  be  instrumental 
in  providing  consistency  in  user  interface  design  decisions  around  NATO. 

2.  The  second  set  of  conclusions  relates  to  the  future  of  Human  Factors  of  Uninhabited  Systems  as  a 
research  field.  The  workshop  sessions  on  highest  priority  research  needs  to  enable  future  NATO  UMV 
visions  revealed  a  wealth  of  ideas.  But  the  main  outcome  for  me  is  that  the  difference  between  the 
roles  of  human  and  machine  as  well  as  the  difference  between  the  notions  of  uninhabited  systems  and 
other  automated  process  will  blur.  Simply  put,  in  the  next  future  we  need  not  focus  much  more  on 
Supervising  Uninhabited  Systems  but  rather  on  the  multi-faceted  and  reciprocal  interaction  between 
human  and  automated  processes.  That  a  particular  system  is  uninhabited  will  be  a  disappearing  notion. 
In  a  sense,  a  future  HFM  taskgroup  might  be  focused  on  Affective  Control  of  Multiple  Human 
Effectors.  But  really,  what  we  need  to  have  is  flexible,  trustworthy  automation  being  operated  under 
user  sovereignty.  This  will  require  not  just  intense  collaboration  but  potentially  even  a  merge  of  the 
field  of  human  factors  engineering  with  other  engineering  fields  such  as  process  automation  and 
computer  science. 
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APPENDIX  A: 

Original  Workshop  Meeting  Announcement  Text 

Supervisory  Control  of  Multiple  Uninhabited  Systems  -  Methodologies 
and  Human-Robot  Interface  Technologies 


A  Workshop  Examining  the  Human  Factors  of  Multi-Uninhabited  Vehicle  Supervisory 

Control 


The  primary  purposes  of  this  Workshop,  which  is  by  invitation  only,  are  to  1)  review  recent  advances  in  the 
areas  of  multi-Uninhabited  Military  Vehicle  (UMV),  supervisory  control  methodologies  and  operator 
interface  technologies,  and  2)  to  identify  the  next  steps  and  critical  research  needed  to  achieve  future  NATO 
visions  for  multi-UV  employment.  A  key  aspect  of  this  workshop  will  be  the  detailed  presentation, 
discussion,  and  critique  of  10  to  12  independent  Technology  Demonstrations  (TDs)  associated  with  UV 
supervisory  control.  These  TDs  have  occurred  within  the  past  three  years  across  eight  NATO  nations  and 
associated  with  NATO  Research  and  Technology  Organization  (RTO)  Human  Factors  and  Medicine  (HFM) 
Technical  Task  Group  170.  Many  of  the  demonstrations  will  be  interactive,  with  time  allotted  to  allow 
informal  detailed  discussions  to  take  place  between  participants  and  demonstration  leads  in  order  to  get  a 
more  complete  understanding  of  the  specific  research  challenges  and  lessons  learned.  Using  knowledge 
gained  from  these  TD  discussions,  this  workshop  will  then  seek  to  identify  additional  research  required  to 
achieve  true  multi-UV  supervisory  control  for  future  NATO  operations.  This  workshop  will  also  identify 
key  programs  and  individuals  (researchers,  engineers  and  scientists)  who  are  contributing  to  the  UV  human 
factors  area  from  NATO  countries  and  it  will  provide  a  valuable  forum  for  the  experts  of  several  countries  to 
measure  progress  in  this  critical  technical  area.  It  will  also  foster  for  the  exchange  of  new  ideas,  concepts, 
and  data  relative  to  the  supervisory  control  of  multiple  semi-autonomous  entities. 

This  workshop  will  bring  together  experts  representing  Uninhabited  Military  Vehicle  (UMV)  operator 
control  station  design,  automation,  adaptive  interfaces,  multi-sensory  interfaces,  decision  support  systems, 
and  networked  operations  to  wrestle  with  unresolved  issues  such  as: 

Supervisory  Control  Frameworks 

Advanced  UMV  Operator  Control  /  Display  Interface  Technologies 
Supervisory  Control  Decision  Systems  Concepts 
Application  of  Adjustable  Levels  of  Automation 
Networked  UMV  Applications  and  Systems  of  Systems 


The  workshop  will  be  unclassified,  and  will  include  overview  presentations,  technology  demonstrations  from 
HFM-170  Task  Group  members,  posters,  and  group  /  workshop  discussions.  All  invitees  are  expected  to 
contribute  through  participation  in  discussion  groups  and  group  reports,  leading  to  the  development  of  an 
advisory  report  to  NATO  RTO. 
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APPENDIX  B: 
Detailed  Workshop  Program 

Monday,  7  May  2012 

•  18:00  -  22:00  (Optional)  Technology  Demonstration  set-up 

Tuesday,  8  May  2012 


•  08.30-09.00 

Welcome  /  Introductions  by  Mark  Draper/Leo  van  Breda 

•  09.00-  10.00 

Keynote  Presentation  :  “ Supervisory  Control  Frameworks:  Cheaper  by  the 
Dozen ”  by  Chris  Miller  &  Gilles  Coppin 

•  10.00-10.30 

Break 

•  10.30-12.30 

Technology  Demonstration  Summaries 

•  CAN-1: 

•  CAN-2: 

•  FRA-1: 

Dr.  Siu  O ’Young-  Multi-crew  Control  of  a  Single  Unmanned  Aircraft 

Dr.  Geoff  Ho  -  Supervisory  Control:  OmniSense 

Prof.  Gilles  Coppin  -  UAV Swarm  Control  -  SMAART Project  "Interacting  with 
multi-agent  systems  /  UA  V swarms  " 

•  FRA-2: 

•  GER-1: 

Dr.  Angelica  Leal  -  PEA  “Human  Factors  &  Authority  Sharing”  (FHPA) 

Prof.  Dr.  Axel  Schulte  -  Cognitive  and  cooperative  assistant  system  for  aerial 
manned-unmanned  teaming  missions 

•  NL-1: 

Dr.  Chris  Jansen  -  Remote  auditory  target  detection  using  an  unmanned  vehicle  - 
comparison  between  a  Telepresence  headtracking  3D  audio  setup  and  a  joystick- 
controlled  system  with  a  directional  microphone 

•  PT-1: 

Mr.  Gil  Gonsalves  -  Supervisory  Control:  Optimal  Distribution  of  Workload  Among 
Operators  for  Mixed  Initiative  Control  of  Multiple  UAVs 

•  SWE-1: 

Dr.  Peter  Svenmarck  -  Task  Switching  for  Multi-UGV  Control ;  Supervisor  Control 
ofUGVs  for  Tactical  Reconnaissance 

•  UK-1: 

•  US-1: 

Mr.  Robert  Taylor  -  Dynamic  Airborne  Mission  Management 

Mr.  Tom  Hughes  -  Multi-UA  V  Supervisory  Control  Interface  Technology >  (MUSCIT) 
Demonstration 

•  US-2: 

•  US-3: 

Mr.  Jay  Shively  -Delegation  control  of  Multiple  Unmanned  Systems  (DELCON) 

Mr.  Michael  Barnes  -  Intelligent  Agents  as  Supervisory  Assets  for  Multiple 
Uninhabited  Systems:  RoboLeader 

•  US-4: 

Dr.  Glenn  Osga  -  Unmanned  Surface  Vehicle  Control  &  Monitoring  Human- 
Computer  Interface  for  Amphibious  Operations 

•  12.30-13.30 

Lunch 

•  13.30-15.00 

Technology  Demonstrations:  Interactive  Demos  and  Critique 
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•  15.00  - 

-  15.30  Break 

•  15.30- 

- 17.10  Technology  Demonstrations:  Interactive  Demos  and  Critique 

•  17.10- 

-17:15  Wrap  Up 

•  17.15- 

-  19.00  Early-Evening  Informal  Social 

Wednesday,  9  May  2012 


•  09.00  - 

- 10.30  Consolidated  Review  and  Discussions  of  Technology  Demonstrations  by  Mr. 

Harry  Funk  &  Mr.  Tim  Barry 

•  10.30- 

-  1 1.00  Break 

•  11.00- 

-  12.00  Keynote  Presentation:  “ Vision  for  Future  NATO  Unmanned  Vehicle  Control  & 

Operations ”  by  Col.  Jeffery  Eggers 

•  12.00- 

-  13.30  Lunch 

•  13.30- 

-  13.40  Charge  to  Work  Groups  by  Dr.  Mark  Draper  &  Mr.  Jay  Shively 

•  13.40- 

- 15.30  Breakout  Group  Discussions:  Priority  Research  Challenges  to  Meet  Future  Vision 

for  NATO  UVs  (WORKING  GROUP  LEADS:  Jasper  Lindenberg,  Ruben 
Strenzke,  and  Angelica  Leal) 

•  15.30- 

-  16.00  Break 

•  16.00- 

- 17.00  Breakout  Group  Discussions  <continued> 

Thursday,  10  May  2012 


•  09.00  - 

- 10.30  Group  Presentations  on  Results  of  Breakout  Meetings  (30  min  each) 

•  10.30- 

-  1 1.00  Break 

•  11.00- 

-  12.00  Research  Priority  Consolidation/Discussion  by  Prof  Gilles  Coppin 

•  12.00- 

-  12.30  TER  Summary  by  Dr.  Hendrik- Jan  van  Veen 

•  12.30 

Closing  Comments  by  Dr  Leo  van  Breda  &  Dr.  Mark  Draper 
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